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(54) A frame-interpolated variable-rate motion imaging system 



(57) A method of recording and processing a digital 
motion image sequence, includes the steps of: record- 
ing a digital motion image sequence at a capture frame 
rate; determining different effective image content 



change rates as a function of the scene content changes 
in different portions of the image sequence; and 
processing the digital motion image sequence to pro- 
duce a processed digital motion image sequence having 
portions with the selected effective change rates. 
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Description 

[0001] The present invention relates to an improved 
system for motion imaging display. In particular, sequen- 
tial image frames are interpolated at a variable rate to 
optimize the motion image quality. 
[0002] Motion imaging systems have been in use for 
over one hundred years. These systems rely upon the 
capture and presentation of a sequential series of still 
images. If the sequence is presented at a sufficiently 
rapid rate, the human eye and brain does not perceive 
the images as separate, still images but rather as a 
smooth sequence of moving images similar to what is 
perceived in the real world. The minimum rate at which 
the images are captured or presented varies somewhat 
in different imaging systems. For example, traditional 
motion picture film is typically acquired and presented 
at 24 frames per second. Standard NTSC television 
presents 30 frames per second while PAL uses 25. Im- 
age sequences are sometimes acquired at one rate and 
presented at another for artistic effect, for example to 
create slow motion effects. 

[0003] Higher presentation frame rates can provide 
more pleasing presentation of changes in scene content 
(such as, pans, object motion, or scene changes) by in- 
creasing the observable detail in the frames, reducing 
blur, and smoothing the motion. Thus, the frame rate 
necessary to maintain the illusion of smooth motion will 
depend on the nature and content of the scene. The sit- 
uation is complicated by the viewer's behavior. For ex- 
ample, the amount and location of detail that is per- 
ceived in a motion sequence depends, at least in part, 
on whether the viewer's eyes track a moving object in 
the sequence. Hence, the necessary frame rate for op- 
timal viewing of a motion image sequence depends on 
the nature of the imagery and the behavior of the viewer 
as well as the viewing environment. 
[0004] Today, motion imaging can be realized with ei- 
ther traditional motion picture films or with electronic im- 
aging devices, either analog or digital. Digital motion im- 
age sequences are very profligate with computing re- 
sources, requiring huge amounts of storage, highrreso- 
lution capture and presentation devices, massive com- 
putational needs, and a large bandwidth for communi- 
cation. Since the amount of image data is dependent to 
a very great extent upon the frame rate, any digital im- 
aging system requires a balance between the amount 
of image data and the perceived motion image quality. 
[0005] The modification of a motion sequence frame 
rate for both capture and display is known in the prior 
art. US Patent 3,723,644 issued March 27, 1973 to 
Haskell et al. describes a system for analyzing the mo- 
tion content of an imaging sequence and adjusting the 
frame capture rate to complement the motion detected. 
Frames are counted and the count used to specify the 
frame repetition rate at playback thus reducing trans- 
mission bandwidth requirements. WO 99/67590 pub- 
lished December 29, 1999 describes varying the pres- 
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entation frame rate of an image sequence to prevent pi- 
racy. Tradeoffs between frame rate and image detail are 
also known. US Patent 5,389,965 issued February 14, 
1 995 to Kuzma describes an image sequence transmit- 

s ted over a fixed bandwidth channel wherein the frame 
rate is increased when image detail is decreased. 
[0006] Image sequences acquired at one fixed frame 
rate can be converted and displayed at another frame 
rate. For example, film-based image sequences ac- 

10 quired at 24 frames per second can be displayed at 30 
frames per second for television broadcast in real-time. 
These conversions are achieved either by replicating 
(repeating) frames or by temporally interpolating new 
frames between existing ones. 

15 [0007] Today, image sequences are captured at a 
fixed frame rate although post-processing can reduce 
the frame rate for compression or communication pur- 
poses. The cost of increasing the capture frame rate is 
prohibitive, particularly for motion picture film, since the 

20 amount of film used is increased according to the in- 
crease in frame rate. Moreover, even if the image se- 
quences are acquired or stored digitally at a higher rate, 
the concomitant increase in computing resources, com- 
munication bandwidth, etc. is problematic. 

25 [0008] There is a need, therefore, for a motion imag- 
ing system that optimizes the motion quality and mini- 
mizes the computing resources needed to store, manip- 
ulate, and communicate a digital motion image se- 
quence. 

30 [0009] This need is met according to the present in- 
vention by providing a method of recording and process- 
ing a digital motion image sequence that includes the 
steps of: recording a digital motion image sequence at 
a capture frame rate; determining different effective im- 

35 age content change rates as a function of the scene con- 
tent changes in different portions of the image se- 
quence; and processing the digital motion image se- 
quence to produce a processed digital motion image se- 
quence having portions with the selected effective 

40 change rates. 

[001 0] The present invention has the advantage of in- 
creasing the motion image quality of a motion image se- 
quence while reducing the communication, computation 
and storage costs for the motion image sequence. 

45 

Fig. 1 is a diagram useful in describing an effective 
image content change rate according to the present 
invention; 

Fig. 2 is a diagram useful in describing an interpo- 
se lated image sequence according to the present in- 
vention; 

Fig. 3 is a diagram useful in describing another in- 
terpolated image sequence according to the 
present invention; 
55 Fig. 4 is an illustration of a prior art image storage 
format; 

Fig. 5 is a diagram useful in describing an interpo- 
lated image sequence including replicated frames 
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according to the present invention; and 
Fig. 6 is a schematic diagram illustrating a system 
useful for practicing the method of the present in- 
vention, 

[001 1 ] The present invention is a method for the pres- 
entation of digital motion image sequences at different 
effective image content change rates (effective rate). 
The effective rate is the rate at which image content 
changes within a portion of ,a motion image sequence. 
Digital motion image sequences are originally captured 
at specific frame rates. The changes within a sequence 
of images determines the effective rate and depends up- 
on the frame-to-frame changes in scene content within 
the image sequence. According to the present inven- 
tion, the digital motion image sequence is first analyzed 
to determine an effective rate and then processed to cre- 
ate a motion image sequence whose image content 
change can be presented at the effective rate. If the dig- 
ital motion image sequence is displayed at a frame rate 
lower than the effective rate, the changes in scene con- 
tent in the sequence will not be smooth and will look 
jerky or jittery. If the digital motion image sequence is 
displayed at a frame rate higher than the effective frame 
rate, the motion in the sequence will be smooth but more 
frames than necessary will be used to reproduce the im- 
age sequence so that digital imaging resources will be 
wasted. 

[0012] The effective rate may be different in different 
portions of an image sequence. In some portions very 
few frames are needed to effectively reproduce changes 
in the scene content. In other portions, many more 
frames will be needed to effectively reproduce the 
changes in scene content. 

[001 3] Referring to Fig. 1 , an example of a 1 0-second, 
240-frame motion image sequence with an original 
frame timeline 10 and an effective frame timeline 12. 
Frames one through 48 can be effectively represented 
with four frames presented over two seconds. Frames 
49 through 120 can be effectively reproduced with 72 
frames over three seconds. Frames 121 through 192 
can be effectively reproduced with 144 frames over 
three seconds. Frames 193 through 240 can be effec- 
tively reproduced with 144 frames over two seconds. 
The total elapsed time for the motion image sequence 
remains the same - ten seconds - but the number of total 
frames in the sequence and the number of effective 
frames per second varies throughout the sequence. The 
effective frame rate 14 varies to effectively reproduce 
scene content changes. In every case, the apparent 
changes in the scene proceed at the rate that they were 
originally captured, i.e. the time scale has not changed. 
[0014] Where the effective frame rate is less than the 
original frame rate, the reduction in frames can be read- 
ily accomplished by deleting frames, by replacing a 
number of frames with an average over multiple frames, 
or by interpolating a representative frame for the multi- 
ple frames in the motion image sequence to create an 



image sequence having the effective frame rate. Where 
the effective frame rate is greater than the original cap- 
ture frame rate, additional frames can be created by 
frame interpolation to create an image sequence having 
the effective frame rate. Frame interpolation is the proc- 
ess of creating new frames within an image sequence 
by analyzing the contents of frames prior and subse- 
quent to the new frame and constructing the new frame 
from the data in the prior and succeeding image frames. 
There are a variety of computational techniques known 
in the art to accomplish this, in particular, the use of mo- 
tion vector analysis and estimation. Preferably, motion 
vector analysis and interpolation is used. For example, 
the Cineon system from Eastman Kodak provides such 
a tool. Alternatively, well known image morphing tech- 
niques can be used to construct interpolated image 
frames. 

[0015] Referring to Fig. 2, a motion image sequence 
20 is originally captured at 24 frames per second. Be- 
cause of the changes in scene content, an effective 
change rate of 48 frames per second is selected for dis- 
play. To create this faster frame rate sequence, the im- 
age sequence 20 is processed. An interpolated frame 
(represented with an asterisk) representing a frame that 
is midway between the original frames is calculated and 
inserted between each pair of original frames to create 
a processed, interpolated image sequence 22. The 
frames of the processed image sequence can then be 
displayed at the effective rate of 48 frames per second. 
[0016] Fig. 3 illustrates a more complex example. In 
Fig. 3 an original motion image sequence 30 is captured 
at 24 frames per second and the effective frame rate is 
determined to be 32 frames per second. The image se- 
quence is interpolated to create a processed sequence 
32 having an effective rate of 32 frames per second, 
however in this example only every third original frame 
is retained while the others are interpolated. Frames 
marked with an apostrophe are interpolated at a point 
three quarters between the two frames, frames marked 
with an asterisk are interpolated at a point midway be- 
tween frames, while frames marked with an ampersand 
are interpolated at a point one quarter of the way be- 
tween frames. 

[0017] Presentation of an effective frame rate motion 
image sequence can be accomplished in two different 
ways. In one embodiment, a presentation device can 
display a frame for an arbitrary amount of time. In this 
embodiment, each frame in the processed digital motion 
image sequence has an associated duration which is 
used to specify the length of time for which the frame is 
presented. This duration can vary from frame to frame 
in the sequence. To accomplish this, each frame in the 
image sequence is accompanied by meta-data informa- 
tion describing the frame rate (or exposure time) for 
each frame in the image sequence. This information can 
be stored in a frame file header or in additional descrip- 
tive information associated with but stored separately 
from, an image frame file. The techniques for storing in- 
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formation with a digital image are very well known in the 
art. Fig. 4 illustrates a storage format 40 for storing the 
images 42 and meta-data 44 in a file header 46 as is 
known in the prior art. 

[0018] In a second embodiment, the presentation de- 5 
vice operates at a fixed presentation frame rate which 
is an integral multiple of any of the effective frame rates 
within the motion image sequence. Each frame has an 
associated replication value representing the number of 
times each frame is to be presented so as to present the 
frames at the effective rate. This technique is illustrated 
in Fig. 5, wherein an original sequence is captured at 24 
frames per second but has effective rates of 24, 32, and 
48 for different portions of the image sequence. The low- 
est common multiple of these effective rates is 96, the 
presentation rate. Each portion is interpolated (as illus- 
trated in Figs. 2 and 3). In the example of Fig. 5, portion 
60 has an effective rate of 24 so each frame is displayed 
four times, portion 62 has an effective rate of 32 so each 
frame is displayed three times, and portion 64 has an 
effective rate of 48 so each frame is displayed two times. 
When stored, replicated frames can be stored multiple 
times or, more efficiently, a meta-data replication count 
can be associated with each frame and used by the 
presentation device. 

[0019] Because the lowest common multiple of the 
various effective frame rates within an image sequence 
can readily become very large, it is expedient to limit the 
number of allowed effective rates (thereby limiting the 
maximum presentation frame rate). This is readily ac- 
complished by rounding an effective frame rate for a por- 
tion of an image sequence to the next highest allowed 
frame rate. While increasing the effective frame rate re- 
quires more frames and image data than strictly neces- 
sary, doing so limits the physical requirements of the 
presentation system and potentially simplifies the sys- 
tem hardware. 

[0020] Fig. 6 illustrates the components of a system 
according to the present invention. These components 
are similar to those used by traditional motion imaging 
film or video systems but the display and computation 
components are capable of working with frames having 
arbitrary time duration or replication counts. In Fig. 6, an 
image recording device 70 such as a film, video, or dig- 
ital camera, stores motion imagery of a scene. The im- 
agery is converted to a digital format if necessary for 
example developing the film in a standard film processor 
(not shown) and scanning the film in a film scanner 72. 
Alternatively the digital motion image sequence can be 
acquired directly with a digital camera 73. The digital 
data representing the motion image sequence is proc- 
essed in a digital computer 76 to create an image se- 
quence having portions with different effective frame 
rates. The processed digital image sequence is stored 
on a storage medium such as an optical disc apparatus 
80, or is communicated through a distribution channel 
82 to a presentation device 84 where it can be presented 
to a viewer. The presentation device will display the im- 



age sequence at a variety of effective frame rates or a 
fixed multiple of the effective frame rates in the image 
sequence. In practice, the presentation device will have 
some practical limitations in frame rate. For example, 
the system will have a maximum rate at which data can 
be read from a storage medium or it may have a fixed 
frame or pixel refresh rate that will limit the maximum 
frame rate achievable. 

[0021] The determination of the effective frame rate 
can be made in several ways. It can be arbitrarily set for 
different portions of an image sequence to a desired rate 
depending upon a priori knowledge of the scene, its 
components, and the desired effect. A second tech- 
nique that might be employed is to automatically ana- 
lyze portions of the image sequence frames to calculate 
the effective frame rate for that portion and use that ef- 
fective frame rate. The frame rate can then change dy- 
namically depending on the image content within that 
portion. 

[0022] The effective frame rate can be modified to 
deal with other issues. For example, the effective frame 
rate might be reduced in response to other limitations in 
the system such as storage or bandwidth or computing 
capability, or to compensate for processing artifacts. Al- 
ternatively, the effective frame rate may be manipulated 
purely for some artistic effect, independently of the op- 
timal viewing or computing concerns. An operator can 
interact with the image sequence to arbitrarily manipu- 
late or select the effective rate. This is illustrated in Fig. 
6 with a conventional computer 76, display system 74, 
interactive devices such as keyboards and a mouse 78, 
and storage device 80. Conventional effects, such as 
slow motion, are readily implemented at this point in the 
process. 

[0023] After the image sequence is processed to op- 
timize the rate for each frame, the sequence is stored 
and communicated using conventional digital storage 
and communication devices, such as magnetic or opti- 
cal media, telecommunication systems, etc. Existing 
techniques, such as compression, may be employed to 
further optimize this process. 

[0024] When delivered to a presentation venue, the 
processed image sequence is displayed. The presenta- 
tion device reads each sequential frame in the motion 
sequence together with the meta-data describing the 
time duration or replication associated with that frame. 
Each image in the sequence is then presented to the 
viewer for the prescribed duration or number of repeti- 
tions. 

[0025] In practice, the interpolated or replicated 
frames can be created in real time within the presenta- 
tion device. In this case, the data describing the desired 
frame rate must be provided to the presentation device 
so that the appropriate interpolated frames can be cre- 
ated. This may reduce the communication bandwidth or 
storage requirement for the image sequence if the ef- 
fective frame rate is anticipated to be faster overall than 
the original. Alternatively, the interpolated frames can 



15 



20 



25 



30 



35 



40 



45 



50 



4 



BNSDOCIO: <EP 1 237370A2_I_> 



7 



EP 1 237 370 A2 



8 



frame interpolation comprises motion vector inter- 
polation. 

6. The method claimed in claim 1 , wherein the step of 
5 determining effective change rates is performed by 

an operator. 

7. The method claimed in claim 1 , wherein the step of 
determining effective change rates is performed au- 

10 tomatically. 

8. The method claimed in claim 1 , further comprising 
the steps of compressing the digital motion image 
sequence after the recording step; delivering the 

is compressed digital motion image sequence to a re- 
mote location; and decompressing the compressed 
digital motion image sequence prior to processing. 

9. The method claimed in claim 1 , further comprising 
20 the step of compressing the processed digital mo- 
tion image sequence. 

10. The method claimed in claim 1 , wherein the capture 
frame rate is a constant frame rate. 

25 

1 1 . The method claimed in claim 1 , wherein the capture 
frame rate is a variable frame rate. 

12. The method claimed in claim 2, wherein the proc- 
30 essed digital motion image sequence includes in- 
formation describing a frame display duration for 
each frame. 

13. The method claimed in claim 3, wherein the proc- 
35 essed digital motion image sequence includes in- 
structions for replicating a frame, and wherein the 
display step further includes replicating frames ac- 
cording to the instructions in a display device. 



be created before the image sequence is to be dis- 
played. This approach will reduce the computational 
burden in the presentation device. 
[0026] An effective frame rate motion imaging se- 
quence can be created and presented to viewers even 
if it was initially acquired at a variable frame rate rather 
than a fixed rate. When processed or manipulated, the 
same process of determining effective frame rates can 
be undertaken; frames are interpolated or removed as 
desired and the appropriate meta-data created and as- 
sociated with the corresponding frames. With variable 
frame rate capture, it is likely that the end result will be 
improved since the variable rate acquisition is likely to 
provide more of the necessary information. 
[0027] The present invention provides a simple way 
to improve motion image sequences for presentation 
quality and storage : communication, or manipulation re- 
quirements. 



Claims 

1 . A method of recording and processing a digital mo- 
tion image sequence, comprising the steps of: 

a) recording a digital motion image sequence 
at a capture frame rate; 

b) determining different effective image content 
change rates as a function of the scene content 
changes in different portions of the image se- 
quence; and 

c) processing the digital motion image se- 
quence to produce a processed digital motion 
image sequence having portions with the se- 
lected effective change rates. 

2. The method claimed in claim 1, wherein the 
processing step includes processing the portions to 
have different display frame rates corresponding to 
the effective change rates and further comprising 
the step of displaying the processed digital image 
sequence at the different display frame rates. 

3. The method claimed in claim 1, wherein the 
processing step includes processing the portions to 
have the same fixed display frame rate and further 
comprising the step of displaying the processed dig- 
ital image sequence at the fixed display frame rate, 
and wherein the display frame rate is greater than 
the effective change rate for at least one portion of 
the digital image motion sequence. 

4. The method claimed in claim 1 , wherein the step of 
processing includes one or more processing steps 
selected from the group comprising frame replica- 
tion, frame interpolation and frame deletion. 

5. The method claimed in claim 4, wherein the step of 



40 14. The method claimed in claim 2, wherein the proc- 
essed digital motion image sequence includes in- 
structions for interpolating between frames and 
wherein the display step includes interpolation 
frames according to the instructions in a display de- 

45 vice. 

15. The method claimed in claim 3, wherein the proc- 
essed digital motion image sequence includes in- 
structions for interpolating between frames and 

50 wherein the display step includes interpolating 
frames according to the instructions in a display de- 
vice. 

16. The method claimed in claim 1, wherein the 
55 processing step is performed by a digital computer. 

17. The method claimed in claim 9, wherein the com- 
pression is MPEG compression. 
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18. The method claimed in claim 1 , wherein the record- 
ing step includes capturing a motion image se- 
quence on silver halide film, developing the film, 
and scanning the film to produce a digital motion 
image sequence. 5 

19. The method claimed in claim 1 , wherein the record- 
ing step includes capturing a motion image se- 
quence with a video camera and digitizing the cap- 
tured sequence. 10 

20. The method claimed in claim 1 , wherein the record- 
ing step includes capturing a motion image se- 
quence with a digital camera. 
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